Thesis “ Architectures for Embedded Multimodal Sensor Data Fusion Systems in the

of the PhD‐Thesis “Architectures for Embedded Multimodal Sensor Data Fusion Systems in the Robotics – and Airport Traffic Surveillance – Domain” submitted by Janis Schönefeld In this work an architecture for embedded multi sensor data fusion is proposed that allows the application tracking algorithms in areas requiring an efficient reliable hardware tracking solution, like airport ground traffic surveillance, autonomous driving and driver assistance technology, small autonomous robots, and indoor pedestrian tracking. In front of this background a hardware architecture for a particle filter based tracking system was developed. A working prototype has been realized as a proof of concept. The tracking hardware developed in this thesis supports the usage of domain specific state transition probability density functions as well as domain specific sensor characteristics. These can be transmitted to the hardware in addition to sensor data to enhance the performance in terms of accuracy of the results computed by the hardware. The result has been reached by deriving a suitable general applicable mathematical model that allows the usage of domain specific concepts such as the movement of a vehicle through the parameterizations of the mathematical models. A distributed hardware software architecture for tracking of multiple objects has been developed. The architecture supports the embedding of the particle filter based tracking system into superordinate systems from the robotic, automotive an airport domain. The software is made up of a lightweight template based C++ library. As part of the distributed hardware software architecture a formal interface for easy usage of the hardware has been developed. The formal interface allows the convenient integration of the hardware in superordinate hardware and software systems. The provision of a Ethernet interface by the hardware and the formal communication interface allow the seamless integration into a wide range of systems using common programming languages like Java, Python, C++, or LISP. It has been shown that it is feasible to build a runway incursion alerting system based on localize sensors and localized signals. It has been shown that runway incursion alerting systems based local surveillance can handle a wide range of runway incursion scenarios but need additional data sources to incorporate information about landing aircraft to provide full cover of reasonable runway incursion scenarios, and that it is possible to push the functional safety level of airport taxi, landing and take‐off operations, with respect to runway incursions, to a safety integrity level (SIL) of at least SIL 1. A comparative simulation study based on the performance of sensors from the automotive field and off the shelf components showed that it is possible to significantly boost the performance of systems that strongly depend on global sensors by the integration of local sensors. The significance of the performance boost is shown by the range of new runway incursion scenarios that can be handled. The performance has been demonstrated by simulating the system with runway incursion incidents that happened in the past. The superior performance encouraged the development of a design for a local runway incursion alerting system based on local sensors and off‐ the‐shelf components. The design combines a tracking and a line surveillance approach to achieve optimal reliability and maximum safety through a synergetic effect by the fusion of the approaches on the level of object assessment. A prototype of the system design using the particle filter based tracking system has been almost completely developed for the airport environment. At the time of writing the prototype is still missing a component for the interfacing of the signals to be provided by the industrial project partners. The prototypes hardware architecture is ‐ as it’s focus is on research and development ‐ slightly different from the system design, that complies to the constraints of the airport environment.